Papers by Hitesh Laxmichand Patel
RCI: A Score for Evaluating Global and Local Reasoning in Multimodal Benchmarks (2025.emnlp-industry)
Copied to clipboard
Amit Agarwal, Hitesh Laxmichand Patel, Srikant Panda, Hansa Meghwani, Jyotika Singh, Karan Dua, Paul Li, Tao Sheng, Sujith Ravi, Dan Roth
| Challenge: | Existing evaluation methods do not explicitly measure this distinction, hindering effective dataset curation and real-world focused model development. |
| Approach: | They introduce a region-based score to quantify a dataset's reliance on global versus local visual information. |
| Outcome: | The proposed model-based score systematically compares model performance on image patches versus full images to determine if tasks require holistic image understanding or can be solved with partial or localized visual cues. |
CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data (2026.acl-long)
Copied to clipboard
Pedro Ortiz Suarez, Laurie Burchell, Catherine Arnett, Rafael Mosquera, Sara Hincapié Monsalve, Thom Vaughan, Damian Stewart, Malte Ostendorff, Idris Abdulmumin, Vukosi Marivate, Shamsuddeen Hassan Muhammad, Atnafu Lambebo Tonja, Hend Al-Khalifa, Nadia Ghezaiel Hammouda, Verrah Akinyi Otiende, Tack Hwa Wong, Jakhongir Saydaliev, Melika Nobakhtian, Muhammad Ravi Shulthan Habibi, Chalamalasetti Kranti, Carol Muchemi, Khang Nguyen, Faisal Muhammad Adam, Luis Frentzen Salim, Reem Alqifari, Cynthia Jayne Amol, Joseph Marvin Imperial, Ilker Kesen, Ahmad Mustafid, Pavel Stepachev, Leshem Choshen, David Anugraha, Hamada Nayel, Seid Muhie Yimam, Vallerie Alexandra Putra, My Chiffon Nguyen, Azmine Toushik Wasi, Gouthami Vadithya, Rob Van Der Goot, Lanwenn ar C’horr, Karan Dua, Andrew Yates, Mithil Bangera, Yeshil Bangera, Hitesh Laxmichand Patel, Shu Okabe, Fenal Ashokbhai Ilasariya, Dmitry Gaynullin, Genta Indra Winata, Yiyuan Li, Juan Pablo Martínez, Amit Agarwal, Ikhlasul Akmal Hanif, Raia Abu Ahmad, Esther Adenuga, Filbert Aurelian Tjiaranata, Weerayut Buaphet, Michael Anugraha, Sowmya Vajjala, Benjamin L Rice, Azril Hafizi Amirudin, Jesujoba Oluwadara Alabi, Srikant Panda, Yassine Toughrai, Bruhan Kyomuhendo, Daniel Ruffinelli, null Akshata, Manuel Goulão, Ej Zhou, Ingrid Gabriela Franco Ramirez, Cristina Aggazzotti, Konstantin Dobler, Jun Kevin, Quentin Pagès, Nicholas Andrews, Nuhu Ibrahim, Mattes Ruckdeschel, Amr Keleg, Mike Zhang, Casper Rufaro Muziri, Saron Samuel, Sotaro Takeshita, Kun Kerdthaisong, Luca Foppiano, Rasul Dent, Tommaso Green, Ahmad Mustapha Wali, Kamohelo Makaaka, Vicky Feliren, Inshirah Idris, Hande Celikkanat, Abdulhamid Abubakar, Jean Maillard, Benoît Sagot, Thibault Clérice, Kenton Murray, Sarah K. K. Luger
| Challenge: | Language identification (LID) is a fundamental step in curating multilingual corpora. |
| Approach: | They introduce CommonLID, a community-driven, human-annotated LID benchmark for the web domain, covering 109 languages. |
| Outcome: | The proposed benchmark covers 109 languages and shows that existing evaluations overestimate accuracy for many languages in the web domain. |
SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved strong performance on natural language to SQL (NL2SQL) benchmarks, yet their reported accuracy may be inflated by contamination from benchmark queries or structurally similar patterns seen during training. |
| Approach: | They propose a syntactic probing framework for detecting and quantifying such contamination in large language models. |
| Outcome: | The proposed framework generates syntactic variants of test queries for four widely used NL2SQL datasets. |
Aligning LLMs for Multilingual Consistency in Enterprise Applications (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Large language models (LLMs) remain unreliable for global enterprise applications due to performance gaps between high-resource and mid/low-resourced languages . |
| Approach: | They propose a batch-wise alignment strategy that aligns model outputs across languages . this method improves non-English accuracy by up to 23.9% without compromising English performance . |
| Outcome: | The proposed approach improves non-English accuracy by up to 23.9% without compromising English performance, model reasoning, or retrieval quality. |
RECOR: Reasoning-focused Multi-turn Conversational Retrieval Benchmark (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks treat multi-turn conversation and reasoning-intensive retrieval separately, yet real-world information seeking requires both. |
| Approach: | They propose a framework that transforms complex queries into fact-grounded multi-turn dialogues through multi-level validation. |
| Outcome: | The proposed framework outperforms existing systems in a number of domains and can be used to improve multi-turn conversation retrieval. |
AccessEval: Benchmarking Disability Bias in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models exhibit disparities in how they handle real life queries. |
| Approach: | They propose a large-scale benchmark to evaluate large language models across six real-world domains and nine disability types. |
| Outcome: | The proposed model outputs show higher factual error, more negative tone, and increased stereotyping with social perception compared to neutral queries. |
Hard Negative Mining for Domain-Specific Retrieval in Enterprise Systems (2025.acl-industry)
Copied to clipboard
| Challenge: | Existing methods for lexical retrieval struggle due to semantic mismatches and overlapping terminologies, and ambiguous abbreviations common in specialized fields like finance and cloud computing. |
| Approach: | They propose a scalable hard-negative mining framework that dynamically selects semantically challenging but contextually irrelevant documents to enhance deployed re-ranking models. |
| Outcome: | The proposed framework improves on public domain datasets and shows that it is generalizable and ready for real-world applications. |
FlexDoc: Parameterized Sampling for Diverse Multilingual Synthetic Documents for Training Document Understanding Models (2025.emnlp-industry)
Copied to clipboard
Karan Dua, Hitesh Laxmichand Patel, Puneet Mittal, Ranjeet Gupta, Amit Agarwal, Praneet Pabolu, Srikant Panda, Hansa Meghwani, Graham Horwood, Fahad Shah
| Challenge: | Document understanding models require large, diverse, and well-annotated datasets that can cost millions of dollars to collect and maintain. |
| Approach: | They propose a scalable synthetic data generation framework that combines Stochastic Schemas and Parameterized Sampling to produce realistic, multilingual semi-structured documents with rich annotations. |
| Outcome: | Experiments on key information extraction tasks show that the proposed framework improves the absolute F1 score by up to 11% while reducing annotation effort by over 90% compared to traditional hard-template methods. |
Crowdsource, Crawl, or Generate? Creating SEA-VL, a Multicultural Vision-Language Dataset for Southeast Asia (2025.acl-long)
Copied to clipboard
Samuel Cahyawijaya, Holy Lovenia, Joel Ruben Antony Moniz, Tack Hwa Wong, Mohammad Rifqi Farhansyah, Thant Thiri Maung, Frederikus Hudi, David Anugraha, Muhammad Ravi Shulthan Habibi, Muhammad Reza Qorib, Amit Agarwal, Joseph Marvin Imperial, Hitesh Laxmichand Patel, Vicky Feliren, Bahrul Ilmi Nasution, Manuel Antonio Rufino, Genta Indra Winata, Rian Adam Rajagede, Carlos Rafael Catalan, Mohamed Fazli Mohamed Imam, Priyaranjan Pattnayak, Salsabila Zahirah Pranida, Kevin Pratama, Yeshil Bangera, Adisai Na-Thalang, Patricia Nicole Monderin, Yueqi Song, Christian Simon, Lynnette Hui Xian Ng, Richardy Lobo Sapan, Taki Hasan Rafi, Bin Wang, null Supryadi, Kanyakorn Veerakanjana, Piyalitt Ittichaiwong, Matthew Theodore Roque, Karissa Vincentio, Takdanai Kreangphet, Phakphum Artkaew, Kadek Hendrawan Palgunadi, Yanzhi Yu, Rochana Prih Hastuti, William Nixon, Mithil Bangera, Adrian Xuan Wei Lim, Aye Hninn Khine, Hanif Muhammad Zhafran, Teddy Ferdinan, Audra Aurora Izzani, Ayushman Singh, Evan Evan, Jauza Akbar Krito, Michael Anugraha, Fenal Ashokbhai Ilasariya, Haochen Li, John Amadeo Daniswara, Filbert Aurelian Tjiaranata, Eryawan Presma Yulianrifat, Can Udomcharoenchaikit, Fadil Risdian Ansori, Mahardika Krisna Ihsani, Giang Nguyen, Anab Maulana Barik, Dan John Velasco, Rifo Ahmad Genadi, Saptarshi Saha, Chengwei Wei, Isaiah Edri W. Flores, Kenneth Chen Ko Han, Anjela Gail D. Santos, Wan Shen Lim, Kaung Si Phyo, Tim Santos, Meisyarah Dwiastuti, Jiayun Luo, Jan Christian Blaise Cruz, Ming Shan Hee, Ikhlasul Akmal Hanif, M.Alif Al Hakim, Muhammad Rizky Sya’ban, Kun Kerdthaisong, Lester James Validad Miranda, Fajri Koto, Tirana Noor Fatyanosa, Alham Fikri Aji, Jostin Jerico Rosal, Jun Kevin, Robert Wijaya, Onno P. Kampman, Ruochen Zhang, Börje F. Karlsson, Peerat Limkonchotiwat
| Challenge: | Southeast Asia is underrepresented in vision-language research . SEA-VL is an open-source initiative dedicated to developing culturally relevant datasets for SEA languages. |
| Approach: | They propose to use crowdsourced, automated image crawling and synthetic image generation to develop culturally relevant datasets for SEA languages. |
| Outcome: | The proposed datasets capture SEA cultural nuances and contexts better than existing datasets. |
PCRI: Measuring Context Robustness in Multimodal Models for Enterprise Applications (2025.emnlp-industry)
Copied to clipboard
Hitesh Laxmichand Patel, Amit Agarwal, Srikant Panda, Hansa Meghwani, Karan Dua, Paul Li, Tao Sheng, Sujith Ravi, Dan Roth
| Challenge: | Existing evaluation metrics for Multimodal Large Language Models (MLLMs) are inadequate to assess their robustness to irrelevant or distracting visual context. |
| Approach: | They propose a patch-context-robustness index to measure MLLMs' robustness to visual context variations. |
| Outcome: | The proposed score measures the robustness of MLLMs to visual contexts across 15 vision-language benchmarks. |
SpeechWeave: Diverse Multilingual Synthetic Text & Audio Data Generation Pipeline for Training Text to Speech Models (2025.acl-industry)
Copied to clipboard
| Challenge: | Text-to-Speech (TTS) training requires extensive and diverse text and speech data. |
| Approach: | They propose a synthetic speech data generation pipeline that generates multilingual, domain-specific datasets for TTS training. |
| Outcome: | The proposed pipeline generates data that is 10–48% more diverse than baseline across various linguistic and phonetic metrics, along with speaker-standardized speech audio while generating approximately 97% correctly normalized text. |
MVTamperBench: Evaluating Robustness of Vision-Language Models (2025.findings-acl)
Copied to clipboard
Amit Agarwal, Srikant Panda, Angeline Charles, Hitesh Laxmichand Patel, Bhargava Kumar, Priyaranjan Pattnayak, Taki Hasan Rafi, Tejaswini Kumar, Hansa Meghwani, Karan Gupta, Dong-Kyu Chae
| Challenge: | Multimodal Large Language Models (MLLMs) have been a key advance in video understanding but their vulnerability to adversarial tampering remains underexplored. |
| Approach: | They evaluate MLLMs against five prevalent tampering techniques to assess their robustness . they use a tampered video format to examine the vulnerability of ML models . |
| Outcome: | The benchmark evaluates MLLMs against five prevalent tampering techniques based on 19 video manipulation tasks. |
SweEval: Do LLMs Really Swear? A Safety Benchmark for Testing Limits for Enterprise Use (2025.naacl-industry)
Copied to clipboard
Hitesh Laxmichand Patel, Amit Agarwal, Arion Das, Bhargava Kumar, Srikant Panda, Priyaranjan Pattnayak, Taki Hasan Rafi, Tejaswini Kumar, Dong-Kyu Chae
| Challenge: | Large Language Models (LLMs) are increasingly being used for communication tasks across different regions. |
| Approach: | They propose a benchmark to evaluate whether Large Language Models are ethically aligned and can be used in real-world situations. |
| Outcome: | The proposed benchmark evaluates whether LLMs comply with or resist swearing instructions and assesses their alignment with ethical frameworks, cultural nuances, and language comprehension capabilities. |
Do Image–Text Metrics Respect Semantic Invariances? (2026.findings-acl)
Copied to clipboard
Amit Agarwal, Hitesh Laxmichand Patel, Meizhu Liu, Jyotika Singh, Karan Dua, Hansa Meghwani, Matthew Rowe, M. Avendi, Yassi Abbasi, Tao Sheng, Sujith Ravi, Dan Roth
| Challenge: | Reference-free image–to–text evaluators are now standard for scoring image–caption alignment, yet it is unclear whether they respect semantic invariances. |
| Approach: | They propose an invariance probe on five popular evaluators under semantics-preserving perturbations along three axes: spatial edits, object changes, and socio-linguistic framing. |
| Outcome: | The proposed invariance probe shows that spatial edits and simple phrasing changes shift scores by ()6% on average and cause ranking flips in up to (),37% of cases. |